DipTransformation - Enhancing the Structure of a Dataset and thereby improving Clustering

Code, data, results

Authors

Reference

---- (When published)

Code

The source code for DipTransformation is included here and can easily be imported into every Java-Editor und should be executable without any additional tweaking. One might have to add the path of the JdistLib-Library ("jdistlib-0.4.5-bin.jar"), which is alreaded included here.

The JdistLib-Library comes originally from here and is used for the Dip-Test.

The main class is run.java. Everything one needs to know to run DipTransformation should be explained there in the comments.

Real-World Data and Results

The real-world data we used are in the directory "/dataset". The data sets are as follows:

Additional to that are also the DipTransformed data sets in the folder "/dataset". The transformed data set have an added "Dip" after the data set name, so is the transformed "Whiteside"-data set called "2 WhitesideDip".

The Running Example goes by the name of "tha". The DipTransformed version is called "thaDip".

The results for the compared techniques as well as the results for DipTransformation + k-means would be in the directory "\label". But due to the extensive testing, which produced 10.000 such label files, this directory is currently empty. The label files are uploaded as a separate .zip file . Simply copy them into the folder "\label", if you wish to reproduce the NMI values.

For DCN we could not save the labels as a separate file, so the reported NMI-values come from their own internal evaluation.

Mammographic Mass

This UCI data set contains data points with missing entries. These fragmented data points have been removed from the data set. All methods were tested on the cleaned data set.

Another metric: Adjusted Mutual Information

We used the NMI-score as a measure of comparison. We are aware of another frequently used, state-of-the-art metric for evaluating clustering results, called Adjusted Mutual Information (AMI). We have found that the results do not vary to the extent that the "take home message" changes, so we omit AMI results to reduce clutter.